SYMMETRY RESULTS FOR COOPERATIVE ELLIPTIC 
SYSTEMS IN UNBOUNDED DOMAINS 
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Abstract. In this paper we prove symmetry results for classical solutions of 
scmilincar cooperative elliptic systems in R^, TV > 2 or in the exterior of a 
ball. We consider the case of fully coupled systems and nonlincarities which 
are either convex or have a convex derivative. 

The solutions are shown to be foliated Schwarz symmetric if a bound on their 
Morse index holds. As a consequence of the symmetry results we also obtain 
some nonexistence theorems. 



1. Introduction and statement of the results 

In this paper we study symmetry properties of classical C 2 solutions of a semi- 
linear elliptic system of the type 

(1.1) -AU = F(\x\,U) in ft 

where F = (/i, . . . , f m ) : ft x R m — > R m is locally a C 1 '" function, m > 2 and 
ft is either M. N or the exterior of a ball, i.e. ft = 1* \ B where B is the unit ball 
centered at the origin and N > 2. In the second case we also require the boundary 
conditions 

(1.2) U = on 9ft. 

When ft = R N some radial symmetry results for positive solutions of (1.1) have 
been obtained using the classical moving plane method under further assumptions 
on F and/or on the decay of the solutions at infinity ( [4], [1]). 
As far as we know there are no results in the case when the solution changes sign 
or the underlying domain is the exterior of a ball. 

Here we use the approach introduced in [11], [13], [9] (see also [12]) in the scalar case, 
i.e. when (1.1) reduces to a single equation, to prove a partial symmetry result for 
solutions of (1.1) and (1.2) having low Morse index, assuming some convexity on the 
nonlinear term F(\x\,U). This approach is not immediately applicable to the case 
of systems, as explained in [2]. However in [2] and [3] using some new ideas and, in 
particular, considering an auxiliary symmetric system, foliated Schwarz symmetry 
of solutions is proved in the case of rotationally invariant bounded domains, i.e. 
balls or annuli. 

In passing from bounded to unbounded domains new difficulties arise, some of which 
are peculiar of the vectorial case and do not appear in the scalar case. Therefore, 
though our strategy is mostly based on combining the approaches of [9] (for the 
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scalar equations in unbounded domains) and of [2] and [3] (for systems in bounded 
domains), we need some new devices, in particular we derive some other sufficient 
conditions for foliated Schwarz symmetry (see Section 3). 
To precisely state our results we need a few definitions. 

Definition 1. Let CI be a rotationally symmetric domain in M. N , N > 2. We say 
that a continuous vector valued function U = (ux, ■ ■ ■ ,u m ) : Cl — » R m is foliated 
Schwarz symmetric if each component Ui is foliated Schwarz symmetric with re- 
spect to the same vector p £ M . In other words there exists a vector p £ M. N , 

\p\ = 1, such that U(x) depends only on r = \x\ and 9 = arccos • pj and U is 

(componentwise) nonincreasing in 9. 

Next we define the Morse index for a solution U of (1.1) and (1.2). To this aim we 
denote by Qu(~i the quadratic form corresponding to a solution, i.e. 



. m rn „ ,, 

(1.3) Qu{M~ / [El V ^! 2 - E 



1 " 3 



dx 



where ip = (ipi, . . . ,ip m ) £ Cl(n,M. m ), i.e. is compactly supported in J7. Through- 
out the paper we will frequently use the fact that Qu is also well defined on functions 
in Ho(£l,M. m ) which vanish outside a compact set. 



Definition 2. Let U be a C 2 (n,R m ) solution of (1.1) and (1.2). We say that 

• U is linearized stable, or it has zero Morse index, if Qu(ijj,^l) > for any 
ip £ Cl(n,M. m ); 

• U has (linearized) Morse index equal to the integer fj, = fJ.(U) > 1, if fi is the 
maximal dimension of a subspace X C C^(f2,R m ) such that Qu(ip,CT) < 
for any tJj £ X \ {0}. 



Finally we recall some coupling conditions for systems. 



Definition 3. • We say that the system (1.1) is cooperative or weakly cou- 

pled in an open set D C CI if 

9fi 

—^(\x\,ui,...,u m ) > for any (x, u X: . . . , u m ) 6 D x M. m 

for any i,j = 1, . . . ,m with i ^ j . 
• We say that the system (1.1) is fully coupled along a solution U in an 
open set D C CI if it is cooperative in D and, in addition, for any I, J C 
{1, . . . ,m} such that 7^0, J / 0, I D J = 0, ID J = {1, ...,m} there 
exist io £ I , jo £ J such that 

{x£D: ^(\ x \,U(x)) >0}) >0. 

Let e £ S' Ar_1 be a direction, i.e. e £ R N , |e| = 1, and let us define the set 

(1.4) f2(e) = {x £ Cl : x ■ e > 0}. 

Our main symmetry results are contained in the following theorems 
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Theorem 1.1. Let U be a solution of (1.1) and (1.2) such that |VJ7| <E L 2 (Q) and 
Morse index fi(U) < N. Assume that: 

i) TTie system is fully coupled along U in Q(e) for any e G S N ~ X . 

ii) For any i,j = l,...,m ^f-(\x\,ui, . . . ,u m ) is nondecreasing in each varia- 
ble Uk, k = 1, . . . , m for any i£fl. 

iii) Ifm>3, for any i G {1, .. . ,m}, fi(\x\,ux,. . .,u m ) = Y%=l 9ik(\x\,Ui,u k ) 

where g lk G C x,£ *([0, +oo) x R 2 ). 
T7ien t/ is foliated Schwarz symmetric. 



Theorem 1.2. Let U be a solution of (1.1) and (1.2) suc/i i/iai |VC/| £ L 2 (f2) and 
Morse index n{U) < N — 1. Assume that: 

i) TTie system is fully coupled along U in fi. 

ii) For any i,j = l,...,m the function ^-(\x\, U) is convex in U : 

^-Qx\,tU( Xl ) + (1 - t)C/(x 2 )) < t^(M,^(a:i)) + (1 - *)^(W. ^(^)) 

for any x,Xi,X2 G and /or any t G [0, 1] . 
Then U is foliated Schwarz symmetric. 

Theorem 1.1 extends the main result of [2] to unbounded radial domains, while 
Theorem 1.2 extends the symmetry result of [3] to the same unbounded domains. 

Let us note that our results do not require solutions to be bounded and neither to 
belong to H^(Q), but only that |VJ7| G L 2 (0). 

The proofs of Theorem 1.1 and 1.2 are technically quite complicated. However 
we want to point out that a crucial point is to have suitable sufficient conditions 
for the Schwarz symmetry, namely the ones contained in Section 3, in particular 
Proposition 3.9. 

Moreover, in the case of Theorem 1.1 and 1.2, to bypass the difficulty of dealing 
with a nonselfadjoint linearized operator we use, as in [2], the linear operator asso- 
ciated with the symmetric part of the jacobian matrix Jf of F which is self adjoint 
and to which the same quadratic form (1.3) corresponds. 

Note that if the system is of gradient type, i.e. F = grad(g), for some scalar func- 
tion g (see [5]) then the quadratic form corresponds to that generated by the second 
derivative of a suitable associated functional and hence the linearized operator is 
sclfadjoint. However this is not the case for many interesting systems as the so- 
called "hamiltonian-systems" ( [5], [2]). 

The two above symmetry theorems can be applied to different kind of systems and 
solutions. In the first one, the hypothesis ii) which implies that each /; is convex 
with respect to each variable Uj, i,j = l,...,m, seems, in some cases, suitable for 
positive solutions. Moreover Theorem 1.1 applies to solutions with Morse index up 
to the dimension N and, for m > 3, requires an additional hypothesis. 
Instead Theorem 1.2 applies more generally to sign changing solutions and does not 
need extra-assumptions for m > 3. On the contrary the hypothesis on the Morse 
index is more restrictive since it requires n(U) < N — 1. 
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As a consequence of the proofs of the symmetry results we derive a necessary condi- 
tion to be satisfied by a solution which could be used to establish some nonexistence 
results 

Theorem 1.3. Assume U is a nonradial solution of (1.1) and (1.2) and either 

a) U has Morse index one and satisfies the assumptions of Theorem 1.1 or of 
Theorem 1.2; 

or 

b) the assumptions of Theorem 1.2 are satisfied and there exist io,jo £ {1, • ■ • , m} 
such that (\x\, S) satisfies the following strict convexity assumption: 

(1.5) ^{\x\,tS' + (1 - t)S") < t^(\x\, S') + (1 - t)^(\x\,S") 

for any t € (0, 1), whenever x € fl and S' = (s' 1; . . . , s' m ), S" = (s'{, . . . , s^J £ 
W. m satisfy s' k ^ s' k for any k <E {1, . . . , to}. 
Then necessarily 

j = l J J = l 

for any i ~ 1, . . . , m, with (r, 9) as in Definition 1. In particular, if to = 2, from 

(1.6) we derive that 

(1.7) l^(N, U(x)) = |^(N, U(x)), for any x en. 

OU2 OUi 

The symmetry results of the previous theorems hold, in particular, for stable solu- 
tions of (1.1) and (1.2). However for these solutions (as in the scalar case) we easily 
obtain the radial symmetry without requiring any hypothesis on the nonlincarity. 
Moreover, if D, = M. N in the autonomous case we get that stable solutions must be 
constant. 

Theorem 1.4. Every linearized stable solution of (1.1) and (1.2) such that \VU\ £ 
i 2 (0) is radial. If, in addition, — M. N and F = F(U), i.e. F does not depend 
on \x\, then U is constant. 

Remark 1.5. This result is analogous to the one for scalar equations obtained 
in [9] . We observe that our definition of linearized stability is stronger than the one 
used in [8] to get a nonexistence result for Hcnon-Lanc-Emdcn systems. 

Finally, as corollary of the symmetry theorems we get some nonexistence results, 
analogous to those obtained in the scalar case (see [9]), but under the stronger 
assumptions of Theorems 1.3. 

Theorem 1.6. Assume that one among assumptions a) and b) of Theorem 1.3 
holds. If ft — M. N and F = F(U), i.e. F does not depend on \x\, then there are no 
sign changing solutions of (1.1) such that 

U(x) — > as \x\ — > +oo. 

Theorem 1.7. Assume that one between assumptions a) and b) of Theorem 1.3 is 
satisfied. If Vt = M. N \ B and F = F{U), i.e. F does not depend on \x\, then there 
are no positive solutions of (1.1) (1.2) such that 

U(x) — > as \x\ — > +oo. 
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For systems of gradient type the previous results can be improved in the sense that 
they hold under the assumptions of Theorem 1.1 or Theorem 1.2. 

Theorem 1.8. Assume that either the assumptions of Theorem 1.1 or of Theorem 
1.2 hold. If the system (1.1) is of gradient type and F does not depend on \x\, then 
there are neither sign changing solutions of (1.1) in M. N nor positive solutions of 
(1.1) and (1.2) in R N \ B such that 

U(x) — ?► as \x\ — > +oo. 

We observe that the case of the system of gradient type is easier. Indeed for 
this kind of systems the existence of a positive solution of the linearized equation 
ensures, as in Lemma 2.1 of [9], the positivity of the quadratic form associated to it, 
and, in some sense, the validity of the maximum principle. This is not true anymore 
for systems which are not of gradient type. In the case of bounded domains (see [2] 
and [3]), to overcome this difficulty we used the principal eigenvalue which would 
not help for unbounded domains, while here we use a new sufficient condition for 
the foliated Schwartz symmetry 

We conclude with a few remarks on the range of applicability of our theorems. 
All our results require some information on the Morse index of the solution. If 
the system is of gradient type, as recalled, the quadratic form corresponds to that 
generated by the second derivative of a suitable associated functional. So, often, 
variational methods, used to find solutions, also carry information on the Morse 
index (see [5]), (see [10], for an example). A standard case is given by solutions 
obtained by the Mountain Pass theorem. 

If the system is not of this type it could happen that the second derivative of 
functionals associated to the variational formulation of the system are strongly 
indefinite. 

As explained and showed in [2] this does not mean that solutions do not have finite 
(linearized) Morse index. We believe that, in general, more investigation about 
stability properties of solutions of systems should be done. 

We also think that the result of Theorem 1.3 is interesting and could give some 
new understanding of systems which have or do not have solutions and are not of 
gradient type. 

The outline of the paper is the following. In Section 2 we state or prove some 
preliminary results while in Section 3 we show sufficient conditions for the foliated 
Schwarz symmetry. In Section 4 we give the proofs of the two symmetry results 
Theorem 1.1 and Theorem 1.2. Finally Section 5 is devoted to the remaining 
theorems. 

2. Notations and preliminary results 

We fix some general notation. Throughout the paper, Br denotes the ball in 
with radius R > centered at the origin. If D is a domain, we denote by C\ (D, M m ) 
the space of all C 1 -functions from D to W n compactly supported in D. 
We start by recalling some statements about linear systems. 

Let D be any smooth domain of 1* and let A(x) be an m x m matrix defined in 
D, i.e. A(x) = (aij(x))™'- =1 and aij(x) £ L^ C (D). We consider the linear elliptic 
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system 

i.e the system 
(2.8) 



-Aip + A{x)tp = in D 
ip = on dD 



-Aip m + YJjLi a mj {x)ipj = in D 
ipi = ■ ■ ■ = tp m = on dD. 



We say that a function ip € 77q(7), R m ) is a weak solution of (2.8) if 

rn m 

^ V^i • V</>* + ^ aij(x)ipjipi dx = 

for any G (£>, R m ). We will denote by (-, -) L 2 the scalar product in 7 2 (7>, R m ), 
i.e. (ip, 0)1,2 = J D dx and by Wtp ■ V0 = YllLi ^V 1 * ' for any vector 

valued function ip,<j> G C 1 (D,R m ). We recall the definition of weakly and fully 
coupled for this kind of systems. 

Definition 4. The linear system (2.8) is said to be 

• cooperative or weakly coupled in D if aij(x) < a.e. in D whenever 

i + r, 

• fully coupled in D if it is weakly coupled in D and for any I, J C {1, . . . , m} 
such that I,J^$, I fl J = 0, 7 U J = {1, . . . , m} t/iere exists io G 7 and 
jo G 7 swc/i i/iai meas ({x G 7) : cii j (a;) < 0}) > 0. 

For any scalar function g, we set g + = max{g, 0} and g~ = minjg, 0}. Similarly, 
for any vector- valued function W = (iui, . . . , iy m ) we set W + = (wf, . . . , W+) and 
= (wj~, . . . , w m ). Here and in the sequel, inequalities involving vectors should 
be understood to hold componentwise, for example ip — (^lj • ■ • V'm) > means 
ipi ~> for any index z = 1, . . . , m. 

We recall some known facts about Maximum Principle for systems, see [5], [6], [15] 
and reference therein for more details. 

Theorem 2.1 (Strong Maximum Principle and Hopf's Lemma). Suppose that the 
linear system (2.8) is fully coupled in D and U = (ui, . . . ,u m ) G C^TJjR 1 ™) is a 
weak solution of (2.8). If U > in D, then either U = in D or U > in D. 
In the latter case if P G dD and U(P) = then ^(P) < 0, where v is the unit 
exterior normal vector at P. 

Definition 5. We say that the maximum principle holds for the operator — A+A(x) 
in an open set D C O if any U G H 1 (D,W n ) such that U < on 3D (i.e. 
U+ G 77 1 (7),R m ) ) and such that -AU + A(x)U < in D (i.e. f D Y^Li Vu > ' 
\7ipi + X)I™ =i o,ij{x)ujtpidx < for any nonnnegative ip G 77Q(7),R m ) ) satisfies 
U < a.e] in D. 

Theorem 2.2. There exists 8 > 0, depending on A(x), such that for any subdomain 
D C the maximum principle holds for the operator — A + A(x) in D C fl provided 
meas(D) < S. 
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We refer to [2] for a general formulation and a proof of Theorem 2.1 and Theorem 
2.2. 

Given the linear system (2.8) in D C O, we can associate with it the quadratic form 

r, ra ra 

(2.9) QaW,D) := / J]|V^| 2 + £ a. l3 {x)^ 3 dx 

i=l i,3=l 

for any -0 £ C^(D,R m ). By density, the quadratic form is well defined also if 
ip is in HQ(D,W n ) and vanishes a.e. outside a bounded set. 
We now consider the (symmetric) linear system defined by: 

-A^i + \ J2™=i ( a ij( x ) + a jii x )) i>j = in D 



(2.10) 



-Aip m + \ Y%Li ( fl ny (x) + a jm (x)) tpj =0 in D 
0i = ■ • ■ = Vm = on dD, 

and observe that to (2.8) and (2.10) corresponds the same quadratic form (2.9). 
Obviously if the matrix A{x) of (2.8) is symmetric then (2.8) and (2.10) coincide. 
Moreover we remark that if (2.8) is weakly (fully) coupled in D then the same holds 
for the symmetric system (2.10). 

Next we consider the bilinear symmetric form associated to (2.9): 

/tci ^ m 

• + - J2 ( a io( x ) + aji{x))) i>^i dx 

for any ip,cf) G C^(D,R m ) (but also for ip,(f) G H^(D,W m ) which vanish outside a 
bounded set). Note that if the quadratic form (2.9) is positive semidefinite then 
the bilinear symmetric form (2.11) defines a scalar product and hence the Cauchy- 
Schwarz inequality holds, i.e. 

(P A (ip, 0, D)f < P A (j,, 0, D)P A {<f>, (f>, D) = Q A {i>, D)Q A (<t>, D) 

for any i/j,<p G Hq(D, M. m ) vanishing a.e. outside a bounded set. 
If D is bounded then the symmetric system (2.10) has a sequence of Dirichlet 
eigenvalues A& such that Afc +oo as k — > +oo and a corresponding sequence of 
eigenfunctions Z k G i?g(Z?,K m ), i.e. functions that satisfy 

-AZ* + i \ Mx) + aji (x))) Z k = \ k Z? in D 
Z k = on dD. 

We denote by X s k (L A ,D) these symmetric eigenvalues. 

The first symmetric eigenfunction, i.e. the function associated with \\{L A , D) 
does not change sign in D and the first eigenvalue is simple, i.e. up to a scalar 
multiplication there is only one eigenfunction corresponding to Af (L A , D). 
See [2] for a careful study of the properties of these symmetric eigenvalues. 



Lemma 2.3. Let L A and Q A be defined as above. If 

(2.12) inf QM, 0(e)) < for any e G S 

*/>eCi(n(e),R"») 



N- 



with Q(e) as in (1-4), then there exists R > (depending on A(x)) such that, for 
any e G S N_1 and for any R> R, the first symmetric eigenvalue \\{L A , fl(e)DBji) 
of the operator L A in the domain BRC\fl(e) with zero Dirichlet boundary conditions 
is negative. 
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Proof. Arguing by contradiction we assume there exists a sequence of directions 
e„ G S ,JV_1 and a sequence of radii R n —> +00 such that 

(2.13) Xl(L A ,Q(e n )nB Rn ) >0. 
Up to a subsequence e„ — > e G S ,JV_1 and it holds 

(2.14) inf Q A (V>, 17(e)) >0 

V>eci(n(e),K«) 

contradicting (2.12). Indeed, if (2.14) does not hold there should exist a func- 
tion V G C£(fi(e),K m ) such that Q A (V>, 17(e)) < and this would imply that 
Af (Xaj 17(e) l~l Bn n ) < for n sufficiently large. The continuity of the first symmet- 
ric eigenvalue implies that Af (La, 17(e„)ri-BR ?l ) < for n large enough contradicting 
(2.13). □ 

We introduce some more notation. For a unit vector e G S ,JV_1 we consider the 
hyperplane T(e) = {x G R , : x • e = 0}. We write a e : 1 — >■ for the re- 
flection with respect to T(e), that is er e (x) = x — 2(x ■ e)e for every x G 17, and 
denote by U 17 ' = {u^',. . . , <„ c ) the function U o cr e . Note that T(-e) = T(e) and 
17(— e) = <r e (17(e)) = — 17(e) for every e G S^ -1 . Finally for a given direction 
e G S^ -1 let us denote by W e = (wi, . . . ,w m ) the difference between U and its 
reflection with respect to the hyperplane T(e), i.e. W e (x) = U(x) — U(cr e (x)). 
Obviously the function W e satisfies the linear system — AW e = F(\x\, U)—F(\x\, U a ' 
in 17 and in 17(e). This system can be written as a linear system in many ways. 
Indeed we need at least two different formulations of it to deal with the different 
hypotheses of Theorem 1.1 and Theorem 1.2. 

First, it is standard to see that the function W e satisfies in 17(e) and in 17 the 
system 



(2.15) 



-AW e + B e {x)W e =0 in 17(e) 
W e = on 517(e) 



where B e (x) = (6| J (x))™. =1 with 

(2.16) b e ij {x) = - ^-(\x\,tU(x) + (l-t)U(a e (x))dt. 

Now, we can write 

/ 4 (|x|,[/(x))-/ ? ;(|x|,?7(a e (x))) 

= f i (\x\,u 1 (x),...,u m (x)) ~ fi(\x\,u1'(x),u 2 (x),...,u m (x)) 

+ fi(\x\,u°°(x),U2(x),.. .,u m (x)) - /j(|x|,u^(x),M2 c (x), . . . ,u m (x)) 



+ fi(\x\,ul B (x),. 
= - (iti(x)wi + 

where 
(2.17) 

%( x ) = ~ j f~f(M> u i"'-- ■, u j-i,~ tu j( x ) + (1 - t)uj{(ye{x)),u j+u . . .,u m ) dt. 



,U m e _ 1 (x),M m (x)) - f l {\x\,U 1 '(x), . . . ,U%(X)) 
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This implies that the function W e satisfies in f2(e) and in VI the linear system 

/ 21r n / -AW e + B e (x)W e = infi(e) 

[ ' \ W e = on dfi(e) 

where B e (x) = (a;)) . . • 

We collect some properties of these linear systems in the following lemma. 

Lemma 2.4. Let U be a solution of (1.1) and (1.2) and e any direction, e £ S . 

i) Assume that hypotheses i),ii) and in) of Theorem 1.1 hold. Then, for any 
x £ SI, we have 

(2.19) bUx)>-^(\x\,U(x)) tfui(s) ><•(*), 

(2.20) 6£(a:) > - ^(|:r|,[/(z)) ^ui(i) > % (z) > u°'(x) 

in particular, ifui{x) = w^ c (a;), Uj(x) = u^(x) thenbfj(x) = —§{^-{\x\,U(x)). 
Moreover the system (2.18) is fully coupled in fl and in fi(e). 

ii) If the system (1.1) is fully coupled along U inQ then the linear system (2.15) 
is fully coupled in f2 and in f2(e) for any e £ S N ~ l . If also hypothesis ii) 
of Theorem 1.2 holds, and we let, for any direction e £ S N ~ X 

(2-21) b^(x) = \ f^-(\x\,U(x)) + ^-(\x\,U(a e (x))) 

then, for any i,j = l,..., m and x £ £1, we have 

(2.22) b%{x)>b1f{x) 
and the linear system 

(2.23) -Aijj + B e ' s (x)ijj = 

is fully coupled in Q and in fl(e) as well for any e £ S 1 ^^ 1 , where B e,s (x) = 
(bif(x)) . . Finally, ifU is symmetric with respect to the hyperplane T{e) 

then 6?.(x) = b\f(x) = §^(\x\,U(x)) for any i,j = l,...,m. 

The proof of this lemma is the same as in the case of bounded domains, see Lemma 
3.1 of [2] for case i) and Lemma 3.1 of [3] for case ii). Throughout the paper we 
will denote by 



„ m m 

(2.24) Qes{^,D) := / ]T|V^| 2 + £ b'rf '(x)^ dx 

<=i t,j=i 

the quadratic form associated with the linear system (2.23), and by 



the corresponding bilinear symmetric form. 
Now we recall the following result. 
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Lemma 2.5. Assume that \VU\ G L 2 (0). Then, for any e G S 1 and for any 
j = 1, . . . , m, we have 

(2.26) [ w] dx^O as R ->■ oo 

^ JB 2n \B R 

where Wj are the components of the function W e . 
See Lemma 2.2 in [9] for a detailed proof. 

Lemma 2.6. Let U be a solution of (1.1) and (1.2), such that the system (1.1) is 
fully coupled along U in 0(e) for any e G S N ~ 1 . Then there exists R > smc/j i/ia£ 
the system (1.1) is /u% coupled along U in 0(e) PI -Br /or any R > R and for any 

eeS N -\ 

Proof. Assume, by contradiction, that there exists a sequence of radii R n —> +oo, 
a sequence of directions e„ G S 1 ^^ 1 and a sequence of subsets /„ C {1, . . . , m} such 
that 

(2.27) meas (L G Sl(e n ) n B Ra , ^-(\x\,U) > Q^j =0 
for any i n G / n and for any j n G {1, . . . , to} \ /„. 

Since /„ C {1, . . . , m} there exists I C {1, . . . , to}, I ^ and a subsequence s(n) 
such that 7 s ( n ) = / for any n G N. Up to a subsequence e s ( n ) — > e G S ,w_1 and 
Cl(e s(n) ) n B Rsin) -+0(e). 

The hypothesis of the fully coupling in 0(e) implies that there exist io G / and 
jo G {1, . . . , to} \ / such that 

x G 0(e),|^(|x|,t/) > o\] > 0. 
duj J / 

Then there exist x G 0(e) andB p (a;) C 0(e) such that B p (x) C |a; G 0(e), > o|. 

By continuity B p (x) C 0(e s ( n )) (~l Br . contradicting (2.27). □ 

Remark 2.7. The same proof of the previous Lemma shows also that, if U is a 
solution of (1.1) and (1.2), such that the system (1.1) is fully coupled along U in 
O then there exists R > such that the system (1.1) is fully coupled along U in 
O n Br for any R > R. Moreover by ii) of Lemma 2.4 we also have that the linear 
system (2.23) is fully coupled in O(e) PI Br for any R > R and for any e G S 1 . 

Let U be a solution of (1.1) and (1.2) and let D C 0. We we will denote by Lu 
the linearized operator at U, i.e. L;y = —A — Jp(\x\,U) where Jp(\x\,U) is the 
jacobian matrix of F computed at U, and we define the linearized system at U, i.e. 



(2.28) 



-Atp- J F (\x\,U)ip inD 
ib = Q on dD. 



Associated with system (1.1), or with system (2.28), we can consider the quadratic 
form 

(2.29) Qu{i>,D):= J |Wi| 2 - £ -^{\x\, U)^ dx 
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and the bilinear symmetric form 
(2.30) 

for any V,<^ € C*(.D,R m ) (but also for ip,<f) € H^(D,W n ) which vanish outside a 
bounded set). In the sequel we will denote by Jp(\x\, U) the transpose matrix of 

M\x\,u). 

For an arbitrary subset D of R w we let Xjj '■ ~^ R denote the characteristic 
function of D. To avoid complicated notation, we denote the restriction of x D to 



arbitrary subsets of again by Xjy ■ 

We introduce a family of cutoff functions which we will be frequently used 
throughout the paper. To this aim we fix a C°°-function £ defined on [0, oo) such 
that < £ < 1 and 

(t)= ( 1 if0<i<l 
tw \ if t > 2. 

For R > 0, we then define 

(2.31) $ R €C™(R N ,R), ^(^ 

We will denote the restriction of £r to arbitrary subsets of M. N again by £r. 



Lemma 2.8. Let U be a solution of (1.1) and (1.2) such that \VU\ £ L 2 (fl). Let 
e G and D C f2(e) 6e a (possibly unbounded) open set such that W^q D = 0. 

Then 

i) if assumptions i), ii) and Hi) of Theorem 1.1 are satisfied and if W e > 
in D, we have 

(2.32) limsu P Q [/ (VF e XDefl,^(e)) < 

where Qu is as in (2.29). 

ii) if assumptions i), and ii) of Theorem 1.2 are satisfied, we have 

(2.33) limsupQ es ((Fy e ) ± XD^,^(e)) < 

i?.->+oo 

where Q es (—,D) is as defined in (2.24). 

Note that since W,% D = 0, the function WxdZr belongs to ff x (17(e), R m ) for 
R > and vanishes a.e. outside a bounded subset of J7(e). Hence the quadratic 
forms Qu and Qes are well defined on these functions. 

Proof, i) The function W e = (wi, . . . , w m ) satisfies the linear system (2.18) in fi(e). 
Multiplying the i-th equation of the linear system (2.18) by WiXD^k and integrating 
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on f2(e) we get 

= / (-Awi)(wiXD£,%) dx + Y] / bij{x)wj(wiXD^,R)dx 

Jfl(e) j=1 Jfl(e) 

= \ Vw, ■W{w l i 2 R )dx + V / b e ij {x)w j (w i ^ R )dx. 

J D j =1 J D 

Using (2.19) and (2.20), we deduce 

0> f Vwi-ViwitDdx-jr [ ^(\x\,U)(w^ R )(w^ R )dx 

■ID „■_, JD ° u 3 



for any i = 1, . . . , m. Letting v R := W c xd£,r, we have, from the definition of Qu, 
that 

Qu{v R ,n{e)) = V / \ywiXD^R\ 2 dx - V / ^-(W, U)(wjXD^R)(wiXD^R) dx 
l=1 Ja(„) iJ=1 Jn(e) <w»j 



= E/ K 2 |V^| 2 + V^-V(^D] dx- I ^(\x\,U)(w^ R )(w^ R )dx 
= J2 Vm i -V(m i eI)-E^(|x|,C/)K^)K^)^ + E / ^l V ^| 2 ^ 

m „ 7U (J f 

<J2 w i\VtR\ 2 dx <Y,~52 w 2 dx^ 0, 

i=1 Jfl i=1 K JB 2R \B R 



having applied Lemma 2.5. Thus i) is proved. 

ii) We give the proof of (2.33) for (W e ) + . The case of (W e )~ follows in the same 
way. The function W e satisfies the linear system (2.15) in f2(e). Multiplying the 
i-th equation of this system by w^xdS, r and integrating on f2(e) we get 



[ (Am,) (wfxDt.%) dx = - f Vw, ■ V (w+&) dx = f >>,,[.r)u- jr; ^,I.r 

Jit(e) JD j=l JD 

for i = 1, . . . , m, so that, letting v R := {W e ) + XD^R, we have 

/ \v VR \ 2 dx = Yl / (wt) 2 m R \ 2 dx- 53 / b%{x) Wo wte R dx. 
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Then, the definition of Q es implies that 

Q es (v R ,Sl(e)) = / \Vv R \ 2 dx+J2 b%(x)wfw+XD&dx 

Me) = i -/n(e) 

m « m „ 

= £ / K + ) 2 |V^| 2 dx- £ / (^.ri!,; ,r, wr/^r/.r 
i=l i,j=l Jd 

m . 

+ £ / bt°(x)wtwfe R dx 
i,j=i jD 

m r. 

= £ / (^w-^w)^ rfi 

i,j=l JD 

+ E / b^( X )w-w+e R dx + J2 K + ) 2 |V^| 2 dx. 

iJ=l J ° i=l jD 

Thus from Lemma 2.4 we get that bfj(x) < for i ^ j and that bfj(x) < bfj(x). 
Then 

Qes (v R , tt(e)) < lim sup — £ / (u;+) 2 cfe 

R->-+cx> i=1 JB 2n \B R 

and (2.33) follows again by Lemma 2.5. □ 

3. Sufficient conditions for foliated Schwarz symmetry 

Lemma 3.1. Let U = (ui, . . . ,u m ) be a solution of (1.1) and (1.2) and assume 
that the hypothesis i) either of Theorem 1.1 or of Theorem 1.2 holds. If for every 
e G S' Ar ~ 1 we have either U > If 7 " in D,(e) or U < If 7 " in Q(e), then U is foliated 
Schwarz symmetric. 

The proof is exactly the same as in the case of bounded domains, see Lemma 3.2 
in [2] and Lemma 3.2 in [3] as well as [16] for the scalar case. 



We will now describe other sufficient conditions for the foliated Schwarz symmetry 
of a solution U of (1.1) and (1.2). 

To this end we begin with some geometric considerations about cylindrical coordi- 
nates with respect to the plane x\X2- 

Suppose (3 G R and let ep = (cos /3, sin /3, 0, . . . , 0) be a unit vector in the x\X2 
plane. Then we consider as before the hyperplane T{ep) and, for simplicity, we will 
use the notations ftp = fl(ep), Tp = T(ep), ap = a efj . 

Using cylindrical coordinates we will write x = (xi, . . . , xn) as x = (r, #,i) = 
(r cos 9, r sin 9, x) where r = y 'x\ + x\, x = (213, . . . ,xn) and 9 G [0, 2ir]. 
It is easy to see that the reflection ap through Tp can be written as 

(3.34) 073(7- cos 6>,r sin 6>,x) = (r cos(2/3 - 9 + tt), r sin(2/3 - 9 + tt), x) 

(in fact 2/3 — 9 + n = 9 + 2(P + ^ — 9 j). It can also be proved analytically writing the 
usual reflection in cartesian coordinates and using simple trigonometry formulas. 
Of course, since the angular variable is defined up to a multiple of 27r, we could also 
write the angular variable of the of the point ap (r cos 9, r sin 9, x) as 2/3 — 9 — tt. 
Let us put hf(6) = 2/3-9±tt. 
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Note that if we choose an interval [Oo,0q + 2tt) to which the angular coordinate 9 
belongs, the images 2(3 — 9 ± tt could not belong to the same interval. Nevertheless 
we observe that for a fixed (3 S M, if we take 9 6 [(3 — -|, (3 + |7r] then h~i(6) belongs 

to the same interval, whereas if we take 9 E [(3 — %tt,(3 + ^] then h^(9) belongs to 
the same interval. More precisely we have 

(3.35) 0e 0-^J+^}^2p-9 + 7r = h+(6) £ tf+^J + lv], 

(3.36) ee[p-^J-^]^2p-e-n = hj(9) E tf-^J+l]. 

This can be easily verified evaluating fv~ in the boundary of the intervals of defini- 
tion, since the mappings h~ are decreasing. 

Let us denote by Ug(r,8,x) the derivative of the function U whit respect to the 
angular coordinate 9. 

Proposition 3.2. Let (3 E R and assume that U is symmetric with respect to the 
hyperplane Ts. If Ug > in fis = [x ■ eg > 0], then for any (3 E \J3 — tt, j3] and for 
any x E Clp — [x ■ ep > 0] we have U(x) < U (ap(x)), while for every (3 E [$, $ + tt] 
we have U > U aft inQp. 

To prove this proposition let us first state and prove a simple lemma. 

Lemma 3.3. Suppose that to E (/3 — \,f3 + ? ] and that the assumptions of Propo- 
sition 3.2 hold. Then 

(3.37) U{r,t,x)>U(r,t ,x) Vi G [t , 2/3 - t + tt]. 

Proof. Since U is symmetric, Ug is antisymmetric with respect to Tg. By hypothesis 
U g > in fts = [x ■ e~ p > 0] = {{r,9,x) : /3-f<6»</3 + f}so that Ug(r,0,x) < 
if /3 + § < 6> < /? + |tt. Moreover by (3.35), if/3-f<i </3 + § then 
ht(to) = 2(3 — to + it E [(3 + -|,/3 + |7r]. This means that U(r, -,x) increases in 

[to, (3 + f], then decreases in [(3 + |, /3 + |tt], in particular in [/3 + ^, 2/3 — to + tt]- 
Since C/(r, i , 5) = f/(r, 2/3 - t + tt, x), (3.37) follows. □ 

Proof of Proposition 3.2. Let (3 E [(3 — it, (3] and let x E Vlp = [x ■ ep > 0], equiv- 
alently x = (r cos 9, r sinfl, J) with /3 — ^ < 6* < /3 + -|. We have to show that 
U(x) <U c p(x)H (3 - § < (9 < /3 + f, (3-tt <(3<(3. 
Let us observe that, since £/ is symmetric with respect to Ts, 

U^{x) = U(o-p(o-p(x))) = U(rcos(9 + 20-(3)),rsm(9 + 20-(3)),x) 

because 2(3 - (2/3 - 9 + tt) + tt = 9 + 20 - (3). 

Let us first assume that x E Qp (~l Q^, i.e. x = (r cos 9, r sin 9, x) with 
(/^-f<)/3-f<6'</3+f(</3+f). Then we can apply Lemma 3.3 taking 
t = 6, t = 9 + 20 - (3): wc have that (3 - § < t = 9 < 8 + 20 - (3) < 2(3 - 9 + tt. 
In fact (3 — (3 > 0, and the last equality is equivalent to 8 < (3 + ^ ), which is true, 
since x E Qp. So from Lemma 3.3 it follows that 

U a 0(x) = U(r,6 + 20 - (3),x) > U{r,9,Z) =U(x). 
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If instead x = (r,6,x) € ft/A^g, i.e. ( /3-§tt < )/3-§ < 6 < /3-§ (< /?+§ ), then 
by (3.36) t := 2/3 - 9 - tt =e \0 - § , j3 + f ] and L/(x) = f7(r, 6», x) = C/(r, t ,x), 
while U a P(x) = U(r,9 + 2((3 — (3),x) so that, as before the inequality follows if 
we show that t = 2/3 - 9 - tt < 9 + 20 - 0) < 2/3 - t + tt = 9 + 2tt. The 
inequality 9 + 2(f3 — f3) < 9 + 2n follows from the relation f3 — tt < (3 < (3, while 
2(3 — 9 — tt < 9 + 2((3 — (3) is equivalent to 9 > (3 — 5, which is true since x E Og. 
Then by Lemma 3.1 also the foliated Schwarz symmetry follows. □ 

Remark 3.4. From Proposition 3.2 it follows that if e is a symmetry direction for 
U and for any other direction e' ^ e in the plane 7r(e, e') generated by e and e', 
using the cilyndrical coordinates with respect to tt, one has Ug > in f2(e), then U 
is foliated Schwarz symmetric, as a consequence of Lemma 3.1. This is the strategy 
of Proposition 3.7 exploiting condition (3.38). 

Proposition 3.5. Let 13 G R and assume that U is symmetric whit respect to Tg. 
Assume further that there exists (3\ < f3 such that for any f3 £ (/?i,/3) we have 
U < U a t in VL [3 . Then U e = % > in f2^. 

Proof. We can write the angular derivative as 

TT , a ~s v U(r,0 + a,x)-U(r,0,x) 

Ug{r,9,x)= hm . 

a^o+ a 

With the change of variable a = 20 — (3), (3 = j3 — we have that (3 — > f3~. If 
a is small then (3 £ ((3i,(3), and, if x S O^, then x 6 Qp definitively for (3 — > (3~ . 
Since U a P (r, 9, x) = U(r, 9 + 2((3 — /?), x) as observed, we obtain that 

TTf a v U(r,9 + 20-l3),x)-U(r,9,i) 

Ug(r, 9, x) = hm 



B (r,9,x) - U( 
0^- 20P-I3) 



l U°e(r,9,x)-U(r,9,x) ^ Q 



□ 



Remark 3.6. Let us remark that if if there exists a direction e 6 iS^" 1 such that 
W e (x) > 0, for any x € f2(e) and if a rotating plane argument can be applied to 
the solution U, then the foliated Schwarz symmetry of U follows from the previous 
propositions. Indeed, if e' is any other direction, e' ^ e, rotating the hyperplanc 
T(e) we get a symmetry hyperplane T(e), with e belonging to the 2-dimensional 
plane generated by e' and e.(see the proof of Proposition 3.9 below for details). 
Then Proposition 3.5 applies and by Lemma 3.1 we obtain the foliated Schwarz 
symmetry of U . 

Let us further notice that in the case of a single equation (m = 1) or of sysyem 
but in bounded domains the rotating plane method can be applied without further 
assumptions, so the previous remark yelds the foliated Schwarz symmetry of U . In 
the scalar case, for bounded domains this was also observed in [14], (Corollary 1.2) 
by a different kind of argument. 

In the case of systems in unbounded domains we need to work under the hypothesis 
of Theorem 1.1 or Theorem 1.2 to perform the rotating plane method. We will use 
this procedure to prove the foliated Schwarz symmetry of the solution of (1.1) and 
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(1.2) in Proposition 3.9. 



We now give another sufficient condition for foliated Schwarz symmetry. 

Proposition 3.7. Let U be a solution of (1.1) and (1.2) such that \VU\ G L 2 (0), 
and assume that the system (1.1) is fully coupled along U in tt. Suppose further 
that there exists e 6 S' JV ~ 1 such that U is symmetric with respect to the hyperplane 
T(e) and 

(3.38) inf QuN)Me)) > 0. 

</>GC c 1 (fi(e),K'™) 

Then U is foliated Schwarz symmetric. 

Proof. First observe that the symmetry of U with respect to T(e) and the coupling 
conditions imply that the system (1.1) is fully coupled along U also in 0(e). 
We want to prove the foliated Schwarz symmetry of U using Proposition 3.2 and 
Lemma 3.1. 

We follow the proof of Proposition 2.5 in [9]. We may assume that e = e-i = 
(0,1,..., 0), so that T(e) = {x £ R w : X2 = 0}. We consider an arbitrary unit 
vector e' e different from ±e. After another orthogonal transformation which 

leaves e2 and T{e2) invariant, we may assume that e' = ep = (cos/3, sin/3, 0, 0) for 
some /3 e (— f j §)• Now we consider the cylindrical coordinates (r, 9, x) introduced 
before. The derivative of U with respect to 6 - denoted by Ug - extended in the 
origin with Ug(0) = if O = K.^, satisfies the linearized system 



(3.39) 



- AUr = J2Y =1 ^(\ x \,U)Ui in O(e) 
Ul = ■ ■ ■ = Ul n = on <90(e). 



All we need is to show that Ug docs not change sign in O(e). Indeed, if this is the 
case, Proposition 3.2 implies that either U < U„ e or U > U a<l in O(e^) and by 
the arbitrariness of ep Lemma 3.1 implies the foliated Schwarz symmetry of U . 
We first note that, by (3.38), the bilinear form Pu{ip, (f>, 0(e)), defined in (2.30), 
defines a (scmidefinitc) scalar product on C^(0(e),R m ) x C\ (O(e), M m ), and the 
corresponding Cauchy-Schwarz-inequality yields: 
(3.40) 

(iVfcM,fi(e))) a < P t/ (^V,fi(e))Pc/(^^0(e)) = Q v ^, 0(e))Q t/ (^ ) 0(e)) 

for any i/j,<j> are iJo(0(e),R m )-functions vanishing a.e. outside a bounded set. We 
consider £r as defined in (2.31) and, for R > 0, we let 

i; K GH 1 (O(e),R m ), v R (x)=Z R (x)U+(x) for any x 6 0(e). 

We also fix <f> 6 C^(0(e),M m ). Then (3.40) yields 

, nM vo/ ■ v* - 1 f (M^, to + ^ (oj) + * 

(3.41) = lim (iV(«fl, 0, n(e)) f < Qu(& 0(e)) limsupQ^, 0(e)). 
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Moreover, since 



/ \Vv R \ 2 dx = J2 \V(t R (Ui) + )\ 2 dx 

Ja(e) i=1 JSl(e) 

m „ m r. 

= E / ((ui) + ) 2 mR\ 2 d X +j2 v(ui) + -v(e R (ui) + ) dx 



I (1(e) i=1 -/fi(e 

from (3.39) we have 



/ |V^| 2 cfe- / |[/+| 2 |V£ fl | 2 da; = / vc/+-v(e 2 ? c/+)dx 
Jn( e ) Jn(e) Jn(e) 

m r. 

= J2 vui-v(e R (ui) + )dx 

i=1 J(l(e)n{U*>0} 

=W [-Au$\(e R (ui) + )dx 

i=1 Ja(e)n{ul>a} 

= E/ E^(N'W(^) + ^^ 



= E / l^(N^) (W) ( U e) + ) dx 

= E / ^(N,tO(&((^) + -(^)-))^«fa: 

and since C/) > for i ^ j, and £r (^0) v h — tncn 

/ \vv R \ 2 dx- [ \u+\ 2 \s7a R \ 2 dx< jr [ ^(\x\,uy R v R d*- 



Then 



n 7YL r\ p 

Qu(v R , {1(e)) = / (|V^| 2 - V -Ji(\x\, U)v j R v R ) dx 



: / K\ 2 M R \ 2 dx<^ f \Ug\ 2 dx 

'Q(e) n JB 2 r\Br 



f \x\ 2 \VU\ 2 dx <A ( |W| 2 a 

R Jb 2 r\B r JB 2r \Br 



Since |VC/| 6 £ 2 (£2) we conclude that lim supj^^ Qu(v R , ^(e)) < 0, so that from 
(3.41) we have 

/ vc/+ • w - \ E (f£(M, co + §£(1*1, CO) « ^ <fc = o. 
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Since <j> £ Cl{Vl{e),W n ) was chosen arbitrarily, we conclude that Ug~ is a solution 
of the symmetric system associated with the linearized system, i.e. Ut is a weak 
solution of 

' -A^^iE-id^d^l-^ + ird^^))^)^ in0(e) 
-A(t/-) + -iEr=i(^(N^) + S:(N^))(^) + = in " (e) 

. U] = ■ ■ ■ = Ul n = on 90(e) 

and, as remarked before, this symmetric system is fully coupled in 0(e). The Strong 
Maximum Principle then implies that either = or > in 0(e) and this 
concludes the proof. □ 

Now we need to recall the following definition 

Definition 6. Let U be a C 2 {Q,W n ) solution of (1.1) and (1.2). We say that U is 
stable outside a compact set K, C O if Qu(i^,^\IC) > for any ip G (f2\/C,R m ). 

Obviously, we have 

Remark 3.8. If U has finite Morse index, then U is stable outside a compact set 
KcO. 

Using the stability outside a compact set, we now derive, by means of a rotating 
plane argument, the following proposition which guarantees the foliated Schwarz 
symmetry of U. 

Proposition 3.9. Let U be a solution of (1.1) and (1.2) such that |V[/| £ L 2 (0) 
and that U is stable outside a compact set /Cell. Assume that one of the following 
holds: 

*) suppose that assumptions i), ii) and Hi) of Theorem 1.1 are satisfied; 
**) suppose that assumptions i), and ii) of Theorem 1.2 are satisfied. 
If there exists a direction e G S 1 ^^ 1 such that 

(3.42) W e {x)>0 or W e {x) < for any x G 0(e), 

then U is foliated Schwarz symmetric. 

Proof. Let us assume that e = (1, 0, . . . , 0) and that W e < in 0(e). We consider 
an arbitrary unitary vector e' G S' Ar_1 different from ±e. We want to show that 
either U > U a "' in O(e') or U < U a "' in O(e'). Then, since the vector e' is arbitrary, 
the foliated Schwarz symmetry follows from Lemma 3.1. 

After an orthogonal change of variable that leaves e = ei invariant we can assume 
e' = ep = (cos /3, sin /3, 0, . . . , 0) for some [3 G (0, tt). We set ep = (cos/3, sin/3, 0, . . . , 0) 
for /3 > 0, so that e = eo- As before we write in short 

0/3 := O(e^) = {x G O : x\ cos /3 + X2 sin/3 > 0} and 
W P . = W e? ^ Tfj ._ T{ep). 
Then we start rotating planes and we define 

(3 = sup{/3 G [0, tt) : W 13 ' < in O^ for all /3' G [0, /3)}. 
Our aim is to show that W 13 = in O^. 

Indeed in this case we can apply Proposition 3.5 getting that the angular derivative 
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U$ in the cylindrical coordinates (r, 6, x) is nonnegative. Then Propositon 3.2 and 
Lemma 3.1 give the foliated Schwarz symmetry of U . 

We observe that, by continuity, < and hence (3 < n, because = -W° > 
in f^Tr = — fio- 

Arguing by contradiction we assume that W p ^ 0. The function satisfies 
both the linear systems (2.15) and (2.18) in fig. Moreover if *) or **) are satisfied 
then these linear systems are fully coupled in fi^ by Lemma 2.4. So by the strong 

maximum principle W 13 < in fis. Moreover, applying the Hopf's Lemma on the 

hyperplane Ts, where W@ vanishes, we have, by Theorem 2.1 

(3.43) — — (x) < for any x G Ts n fi. 

Since, by hypotheses, U is stable outside a compact set, there exists Rq > such 
that 

(3.44) Qu(^,tt\B Ro )>0 forcvcry7/.eC c 1 (fi\S flo ,]R m ). 
We fix Ri > Rq, and we claim that there exists £\ > such that 

(3.45) W^ +£ (x)<0 in B Rl nfi^ +£ Ve G [0,£i). 

In the case fi = \B, let Bs be a neighborhood of dB in fi of small measure to 
allow the maximum principle to hold in Bs for the operator — A + B@ +e {x) in case 
*) is satisfied or for the operator —A + B° +£ (x) in case **) holds, for sufficiently 
small £ > 0, see Theorem 2.2. We first show that 

(3.46) W^ +£ (x)<0 m B Ri n(n^ +£ \B s ) Ve £ [0,ei). 

If (3.46) is not true, we have sequences e n -> and x n G B Rl n (^p +e \ Bg^j such 

that W^ +e " (x n ) > 0. After passing to a subsequence, x n —> x G B Rl PI ^fi^ \ BsJ 

andW^io) = 0, hence x G 1). Since W f3+£ "(x) = on T~ p+£n and W? +en (x n ) > 0, 
there should be points £ n on the line segment joining x n with Ts +e and perpendi- 
cular to Tjj +e , such that ^j— — — L (£n) > 0. Passing to the limit we get ^r-(xo) > 
in contradiction with (3.43). So we get (3.46). 

By the maximum principle, the definition of Bs and (3.46) we get W@ +£ < 0, under 
both assumptions *) and **), also in B Rl (~l Qp +e H Bs and hence (3.45) holds. 
If fi = M. N by the same argument, directly from (3.43) we get (3.45). 
Now we want to prove that 

(3.47) W? +£ <0 infig +e for all £ G [0, £i). 

Because i?o < Ri, by (3.45) the function v := (W^ +£ ) + x„ has its support 
strictly contained in fi \ B Ro . We claim that 

(3.48) v = 0. 

We first consider the case where assumption *) is satisfied. Let <f) G C%°(Qp +e \ 
B Ro ,R m ). By (3.44), the bilinear form P v , defined in (2.30) defines a (semidefi- 
nite) scalar product on C\ (fi \ B Ro , R m ) and also on C\ (fi s +s \ B Ro , R m ) , and the 
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corresponding Cauchy-Schwarz-inequality yields 

<P, % +e \ Sflo)) 2 < ftrty> V>, \ B Ro )Pu{<p, <p, n- p+e \ B Ro ) 

for all ij},(p in i?o(^fl+e \.B# ,R m ) that vanish a.e. outside a bounded set. Conse- 
quently, we obtain 
(3.49) 

(Pu[v R , <j>, fy +e \ BaS) 2 < Qu(vr, fy +e \ B *o) n^ +e \B Ro ) for R > 0, 

where v R = v£, R and £r is defined in (2.31). The function v is nonnegative in Qa +e , 
so we are in position to apply Lemma 2.8, part i), getting from (2.32) 

hmsupQu(v R ,Qs +e \B Ro ) < 0. 

R— >-oo 

Combining this with (3.49), we find that 

f; vt* • v0 2 - If; (J£(w,to + ^(N.to) ^ 

i=l i,j = l \ ■? 



lim P,/(v R , ^JlXBuo ) = 0. 



Since <^> € C£°(f23 +e \B Roi R m ) was chosen arbitrarily, we conclude that v is a weak 
solution of the linear symmetric system 

-At».-|f;(^(N,E0 + ^(N^)^=0 in Q, +e \^ 

for j = 1, ... ,771. Then however u = {W^ +e ) + xn^ +e = by the unique continuation 

principle, since (W^ +e )+ = in £ Rl n VL~ p+e by (3.45) and i?i > i? . Hence (3.48) 
holds. 



Next we consider the case where hypothesis **) holds. Since every function t 6 
C^(n^ +e \B Ro ,W m ) can be extended to an odd function t £ C^(n\B Ra ,R m ) with 
respect to the reflection at T^ +E , we have by (3.44): 

Q w {T,n~ p+£ \B Ro ) = ±Qu(f,n\B Ro ) > 

for all t £ C^(f^ +£ \ B Ro ,M. m ). Hence the bilinear form P e . s associated with 
Q £ p +c s defines a (semidcfinitc) scalar product on C*(fig +e \ B Ro ,M. m ), and the 
corresponding Cauchy-Schwarz-inequality reads 

(Pe 0+eS (v R ,(f>,£lp +s \B Ro )j < Qee +s s(<f>,ttp +e \B Ro )Q e - +eS (v R ,tlp +e \B Ro ) 
(3.50) < Qe s+c s(<PM f 3 +E \B Ro )]imsvpQ e a (v R ,Sls +e \ B Ro ) 

R— ¥+oc 

for Vr := v^ R and for any <j> <G Cl(£ls +E \ -Bjj ,R m ). Using ii) of Lemma 2.8 with 
D = ( g +6 \ B Ro then we have 

UmsupQ ejS+eS (v R ,rip +E \B Ra ) < 0. 

R— > + oo 
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Then, from (3.50) it follows that 

Jtt0 +e \BB o i = l 

for any <p £ C*(£lp +e \ B Ra ,W m ). Then (w^+ 6 ) + is a solution of the linear 
symmetric system 

i m 

-Awf + - £ (V V) + $ + ''(*)) < = , in tt He \ B Ra 

^ 3=1 

for i = 1, ...,m. Then, the unique continuation principle, implies that v = in 
^/g+e \ Br an( i (3-48) holds also in this case. 



As a consequence of (3.48), we have got (3.47). Then the definition of j3 implies 
that W p = in and this gives the claim. 

□ 



4. Proofs of Theorem 1.1 and Theorem 1.2 

Proposition 4.1. Let U be a solution of (1.1) and (1.2) with Morse index n(U) < 
N and assume that the system is fully coupled along U in f2(e) for any e £ S . 
Then there exists a direction e £ S N ~ X such that 

(4.51) Qu(ip,n(e)) > for any $ £ CS(fi(e),R m ). 

Proof. The case /z(f7) < 2 is immediate. Indeed at least one among , f2(e)) 

and Qu{~) ^(~ e )) should be positive scmidcfinitc, otherwise we would obtain a 2- 
dimensional subspace of C^(f2,R m ) where the quadratic form Qu(— , 0) is negative 
definite, contradicting the assumption of Morse index less than 2. 
So let us assume 2 < j := fJ>(U) < N. By definition, j is the maximal dimension 
of a subspace X,- := span{*i, . . . , * 3 -} C C£(Q, R m ) such that Qu(i/),Q,) < 
for any ijj E Xj\ {0}. We take a ball _B P with radius p > sufficiently large 
to contain the supports of all i = 1, . . . , j. For R > p, in the domain O n 
the linearized operator Ljj defined in (2.28) has exactly j negative symmetric 
eigenvalues, Af (Lu, flflBn) < X^Lu, OflBjj) < • •• < A|(L[/, OD-Br) with respect 
to Dirichlet boundary conditions, and Xj +1 (L U7 ft n Br) > 0. See Section 2 for the 
definition of the symmetric eigenvalues and [2] for their variational characterization. 
Now assume, arguing by contradiction, that for any e £ (4.51) does not hold. 

Then, we can apply Lemma 2.3 to the linear operator Lu and we can find a R > 
such that, for any R > R and for any e £ S fJV ~ 1 the first symmetric eigenvalue 
AJ(ij/,0(e) n Br) of the linearized operator in f2(e) (~l Br, with zero Dirichlet 
boundary condition is negative. 

We can take R > max{p, i?, -R} where R is as in Lemma 2.6. In this way we 
have that the linearized system, defined in (2.28), is fully coupled in f2(e) fl Br for 
any e £ S' Ar_1 and the same holds for the symmetric system associated with the 
linearized operator in 0(e) C\Br. We denote by <I> e the first positive L 2 -normalizcd 
cigenfunction of the symmetric system —A — ^(Jp(\x\,U(x)) + J F (\x\, U(x)) in 
il(e) fl Br (we observe that $ e is uniquely determined since the system is fully 
coupled in Q(e) CiBr for any e £ S^ -1 ) and by 3>i, . . . , <&j the mutually orthogonal 
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eigenfunctions corresponding to the j negative symmetric eigenvalues of Ljj in 
fin B R . Define 



$ e (x) if x e 0(e) n Bii 



where (— ,— )l 2 (d) denotes the scalar product in L 2 (D,M. m ). The mapping e i— > 
is a continuous odd function from S^ -1 to i?Q (O, M" 1 ) and by construction 
(* ej $i)L2 ( f2 nSji) = 0. Therefore the mapping h : S^ -1 -> W' 1 defined by 

h(e) = ((*e,*2)£>(nnB«).--- > (*e.*j)i='(nnB ll )) 
is an odd continuous mapping, and since j — 1 < JV by the Borsuk- Ulam Theorem 
it must have a zero. This means that there exists a direction e £ S N_1 such that 
^ e is orthogonal to all the eigenfunctions $i, . . . , <f>j in L 2 (0 n £Jr, R m ). Since 
= j this implies that j§^f^^ > \" j+x {L u , fin Br) > against the fact 

that 

Qu(* e ,n n Bk) = f ( ^: e '^i L2(n( " e)) > ) a? (£t/, 0(e) n b r ) 



A;(Lt,,fi(-e)nBfl) <0 



($ e , $i)i2 (n(e)) 

by construction. The contradiction proves the assertion. □ 
Now we are in position to prove Theorem 1.1. 

Proof of Theorem 1.1. By Proposition 4.1 we have that there exists a direction 
e € S' Ar ~ 1 such that (4.51) holds. Hypothesis i) of Theorem 1.1 implies that the 
system (1.1) is fully coupled along U also in O and so, if W e = in 0(e), we 
immediately get the foliated Schwarz symmetry of U, by Proposition 3.7. 
If instead W e ^ we show that W e is eather strictly positive or strictly negative 
in fi(e) so that, by Proposition 3.9 we again get the assertion. 

From (4.51) we have that Pj/(V>> 4 1 , 0(e)) is a semidefmite scalar product on (0(e), M. m ). 
Consequently, using the Cauchy-Schwarz-inequality we obtain: 

(4.52) (Pu(vr, 0, 0(e))) 2 < Qu(vr, O(e))Q c/ (0, fi(e)) 

for any <j> s C\ (0(e), R m ), where vr := (W e ) + xn(e)£fl an d £,r is a cut-off function 
as defined in (2.31). By Lemma 2.8 we have lim sup i? ^. +oc Qu( v R, 0(e)) < 0. 
Combining this with (4.52) and passing to the limit as R —> +oo we have 

/ V(^ e )+-V0-if] (&\x\,U) + ^(\x\,U))wf<t>idx 

= lim Pu(vR,(t>,n(e)) = 0. 

R— >+oo 

Since <fi £ (0(e), M m ) was chosen arbitrarily, we conclude that (W e ) + is a solution 
of the system 

(4.53) -Awf-±f2(^(\x\,U) + ^(\ X \,U))w+=0 in 0(e) 
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for i = l,...,m. Now, since (W e ) + > in 0(e) and the linear system (4.53) is 
fully coupled in 0(e), the Strong Maximum Principle implies that cither (W e ) + = 
or (W e ) + > in 0(e). In any case W e is strictly positive or strictly negative in 
0(e). □ 

Proposition 4.2. Let U be a solution of (1.1) and (1.2) with Morse index p,{U) = 
j < N — 1 and assume that the system (1.1) is fully coupled along U in CI. Then 
there exists a direction e G S N ~ X such that 

(4.54) Q es (ij, 0(e)) := / | Wf + V b%{x)^^ dx > 

M°) iJ=1 

/or any V £ C^(fl(e),R m ) . 

Proof. Assume, arguing by contradiction, that for any e G (4.54) does not 

hold. Then we can apply Lemma 2.3 to the linear operator L es , defined in (2.23), 
and we can find R > such that, for any R > R and for any e G S' JV_1 thc first 
symmetric eigenvalue Af(L es ,0(e) n -Br) of the linear operator L es in 0(e) n Br 
with Dirichlct boundary conditions is negative. 

By definition j is the maximal dimension of a subspace Xj := span < . . . >C 
C*(fi,R m ) such that Qu(tp,Q.) < for any ip G X, \ {0}. Wc take a ball B p with 
radius p > sufficiently large to contain the supports of all i = For 
R > p, in the domain n -Br the linearized operator has exactly j negative 
symmetric eigenvalues and \j +1 (Lu,Bn f) 0) > 0. 

We take R > max{p, R, R}, where R is as defined in Remark 2.7. In this way 
we have that the linear system —A + B es (x) defined in (2.23), is fully coupled in 
0(e) n Bji for any e G S N ~ X and the same holds for the symmetric system associ- 
ated with the linear operator L es in 0(e) n Br. Moreover the system (1.1) is fully 
coupled along U in fl Br. 

We denote by g e the first positive L 2 -normalized eigenfunction corresponding to 
Af(L es ,0(e) n B R ) (which is defined since the system -AV + B es {x)V is fully 
coupled in 0(e) n Br for any e G S^ -1 ) and by g e the odd extension to On Br. 
Let $i, . . . , $j be the mutually orthogonal eigenfunctions corresponding to the j 
negative symmetric eigenvalues of Ljj in n Br. Let h : S N ~~ 1 — be defined by 

K e ) : = ({9e,$i)L2(nnB R ), ■ ■ ■ ,(g e ,$j)L2(nnB R )) 

where (— , — )l 2 {D) i s > as before, the usual scalar product in L 2 (D,K m ). h is an 
odd and continuous mapping and, since j < N — 1 it must have a zero by the 
Borsuk-Ulam Theorem. This means that there exists a direction e G such 
that g e is orthogonal to all the eigenfunctions $i, . . . , $j in L 2 (0 n Br, R m ). Since 
fJ-(U) = j this implies that 

Qu(g e ,nnB R ) = LnB R l v ^l 2 - T,Z=i §£-(Hu)glg j dx 

{9e, ge)h 2 {nr\B R ) {9e, ffe)z,2(nnS R ) 

The symmetry properties of g e imply that 



> \' j+1 (Lu,SlnB R ) >0. 



/ l V 5e| 2 " E ^{\x\, U)~gl~gi dx = 2 [ |V. 9e | 2 + ]T h%(x)g\g{ dx 

Jn lj=1 OUj Jn{e)nB R lJ=1 

= 2\ s 1 {L es ,n(e)nB R ) < 0. 

The contradiction proves the assertion. □ 
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Proof of Theorem 1.2. From Proposition 4.2 we have a direction e £ S 1 
such that Qes(ip,to(e)) > for any ip £ <_7i(0(e),M m ). If W e = 0, i.e. if U is 
symmetric with respect to T(e), then Q es 11(e)) = <2[/(^, 0(e)) > for any 
■0 £ C\ (0(e), R m ). The foliated Schwarz symmetry of i7 then follows from Propo- 
sition 3.7. 

So assume W e ^ 0. By assumption the bilinear form P es , defined in (2.25), defines 
a scalar product on C£(fi(e), M m ), so, for w fl = (W e ) + £ij £ i#( n ( e )> R ) and for 
any £ C c 1 (0(e),M m ), we have 

(4.55) < (P es (ffl>,^(e))) 2 < Q es (v R ,n(e))Q es (4>,n(e)). 
Moreover, using (2.33) with D = 0(e), we have 

limsupQ es (t!_R,r2(e)) < 0. 

R— »~|-oo 

Passing to the limit in (4.55) we get 

P es ((W e ) + ,<f> 7 n(e)) =0 

for any <f> £ C* (0(e), R m ), so that (W e ) + is a weak solution of 

i m 

(4.56) - A( Wl )+ + - YM^) + = in n(e) 

j'=i 

for i = 1, . . . , m. Since the system is fully coupled in O(e), the strong maximum 
principle implies that either (W e ^j > in 11(e) or (VK 6 ) = in O(e). In any 
case W e is strictly positive or strictly negative in O(e) and the foliated Schwarz 
symmetry of U follows from Proposition 3.9. □ 



5. Other results 

We prove the other theorems stated in Section 1. 

Proof of Theorem 1.3. If U is a Morse index one solution for any direction e £ S 1 ^^ 1 
at least one among Qu(~, 0(e)) an d Qu{~, 0(— e)) should be positive semidefinite, 
otherwise we would obtain a 2-dimensional subspace of C^(0, M m ) where the qua- 
dratic form Q[/(— ,0) is negative defined, contradicting the definition of Morse 
index 1. 

By the proof of Theorem 1.1 and of Theorem 1.2 we can find a direction e £ S fJV ~ 1 
such that W e = in O(e) or W e > in O(e). In the second case, applying Propo- 
sition 3.9 we can find a direction e' such that W e = in O(e'). So, in any case, 
there exists a direction e such that U is symmetric with respect to the hyperplane 
T(e). Thus, by symmetry, Qu(ijj,n(e)) = Qu(ip,fl(-e)) for any ip £ C£(0(e), K m ) 
and Qu is positive semidefinite in O(e). 

After a rotation, we may assume that e = e% = (0, 1, . . . , 0) so that T(e) = {x £ 
I 1 *' : X2 = 0} and we may introduce new (cylinder) coordinates (r, 0, 2/3, . . . , j/jv) 
defined by the relations x = r[cos#ei + sin#e2] + yi— 3 yie^. 

Then the angular derivative Ug of U with respect to 8, extended by zero at the 
origin if O is a ball, satisfies the linearized system, i.e. 

(5.57) - AU 9 - Jf {\x\,U)U e = in 0(e 2 ) 
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and it is zero on the boundary. Reasoning exactly as in the proof of Proposition 
3.7 we have that Ug~ is a solution of 

(5.58) -AU+-^(J F (\x\,U) + J F (\x\,U))U+ = in 0(e 2 ) 

and this implies that Ug does not change sign in 0(e2). We can assume that 
Ug = Ufj~ , and that is a solution of the systems (5.57) and (5.58). Then 
J F (\x\, U)U e = \ (Jf(\x\, U) + J F (\x\,U)) Ug, i.e. (1.6) and if m = 2, since Ug is 
positive, we get (1.7). The case Ug = Ug can be handled in the same way. This 
proves the assertion if the hypothesis a) holds. 

To prove the theorem under assumption b), we observe that the result follows if 
we can find a direction e such that W e = in 0(e) and Qu(ip, 0(e)) > for any 

Following the proof of Theorem 1.2 we have a direction e 6 S N_1 such that cither 
W e = and Q v (ip, 0(e)) > or W e > in 0(e). 

The second case cannot happen. Indeed the function W e satisfies the system 

m 

(5.59) - A Wl + b %{x)wj =0 in 0(e) 

i=i 

where bfj(x) are as in Lemma 2.4. Multiplying the i-th equation of (4.56) and 

(5.59) for Wi£,R (£r is the usual cutoff function, see (2.31)), integrating in 0(e), 
summing on i and subtracting, we get 

E f a( ) ( b U x ) - \(W + ^(^))) WiWitudx = 

equivalcntly 

(5.60) V / (b! j (x)-b%(x))w j w i t R dx = 0. 

Since W e > 0, Lemma 2.4 implies that bf,-(x) > bfj(x) for any i, j = l,...,m, 
and since WiWj^R > in O(e), relation (5.60) gives b\j(x) = &fj (cc) for any i, j = 
1, . . . , m. 

By the strict convexity of we have that, if M^ 6 > then b\ Q j a {x) > &^ o s Jo (x) 
getting a contradiction. Therefore the only possible case is W e = in O(e) and 
Quii'i ^( e )) ^ 0- Then the result follows as in the previous case. □ 

Proof of Theorem 1.4- We choose, as before, the cylindrical coordinates with re- 
spect to the plane X1X2, i.e. (r, 8, x). Again the derivative Ue satisfies the linearized 
system —AUg — Jp(\x\, U)Ug = in O and, in the case O = l w \ Br(0), also the 
boundary conditions Ug — on 90. By the stability assumption, we can proceed as 
in the proof of Proposition 3.7, with O in place of O(e), to show that Ug does not 
change sign in O. Since Ug is 27r-periodic this is impossible and therefore Ug = 0. 
By the arbitrarily of x\, X2 we conclude that U is radial. 

Moreover, if O = and F does not depend on \x\, then for every t the translated 
function U(x + t) is also a stable solution of (1.1), therefore it is radial by the 
argument above. This however is not possible unless U is constant. □ 

Proof of Theorem 1.6. Suppose by contradiction that (1.1) admits a sign changing 
solution U on R N that satisfies the assumptions of Theorem 1.3 and such that 
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lim U(x) = 0. Since we can apply Theorems 1.1 and 1.2 then U is foliated 

\x\— yoo 

Schwarz symmetric. By a rotation of coordinates, we may assume that p = ejv in 
the definition of foliated Schwarz symmetry, so that U is axially symmetric with 
respect to the axis Ke^r and nonincreasing in the angle 9 = arccos • By the 

proofs of Theorem 1.3 we get a direction e G S N ~ X such that U is symmetric with 
respect to T(e) and 

(5.61) inf Qu(<ip,n(e)) > 0. 

Veci(n( e )) 

We may assume that e = ei in (5.61). Indeed, this is clearly possible if U is radial. 
Moreover, if U is nonradial, then U is strictly decreasing in the angle 9, therefore 
the symmetry hyperplanes of U are precisely the ones containing ejy, and for each 
one of them the infimum in (5.61) takes the same value. 

We now consider the derivative which, by regularity theory, (see [9] Lemma 
6.1) belongs to H 2 (il(ei)) and satisfies the linearized system 

(5.62) - A O-^))(£) =0 in ° (ei) - 
Moreover, because of the symmetry with respect to T{e{) we have 

4^ = on an(ei) 

OXi 

so that g i?o(fi(ei)). From (5.61) we have that Pjj defines a (semidefinite) 
scalar product on C\ (fi(ei), M. m ) and the corresponding Cauchy-Schwarz-inequality 
reads 

(5.63) (Pu(v R ,4,,n(ei))f < Qcr(&^ei))limsupQ c ,(t>«,n(e 1 )) 



R-^+oo 

\ + 



if v# = ( tP^- ) with £r being the usual cutoff function. Reasoning exactly as 



in the proof of Proposition 3.7, with (j%j^J instead of Uq~, we get 

Qu(v R ,tt(ei)) < [ (^-) 2 \V£ R \ 2 dx<-^ [ \VU\ 2 dx 
Jate,) \dxxj R Jb 2R \b r 



so that 



limsupQiy(uR, f2(ei)) < 0. 

i?-> + oo 



Then (5.63) implies, as before, that (§^\ is a weak solution of the system 

(5.64) -a(J^) -l{MU(x)) + Jl(U(x)))(^-\ =0 inf2( ei ). 

Since this system is fully coupled in fi(ei) the Strong Maximum principle implies 
that either (^§§^j = in O(ei) or > in f2(ei). In any case does not 

change sign in fi(ei) which is impossible because U changes sign in fl(e\) and 
lim U(x) = 0. This contradiction proves the assertion. 

□ 
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Proof of Theorem 1.7. Suppose by contradiction that (1.1) and (1.2) admit a pos- 
itive solution U in M. N \ B that satisfies the assumptions of Theorem 1.3 and such 
that lim U (x) = 0. As in the proof of Theorem 1.6 we may assume that U is sym- 

|a;|— ^oo 

metric with respect to T{e\) and that (5.61) holds for e = e±. Then the derivative 
satisfies the system 

- in n(eij 

on T(e x ) HO 



(5.65) 



V OX I 

1^=0 



jx i 



on dB n 0(e a 



Multiplying by (^§^j integrating over O(ei) and using the cooperativeness of 
(1.1), we get 



y I 



•V 



duj 



dx < 



E 



l)J=1 ^n(-ei) ""J 
dU 



duj 



Then, as in the proof of the previous theorem, we get that (-§^)~ is a solution of 
the symmetric system (5.65) and hence (J§^) _ = by the boundary conditions. 
Then > in Cl(e\) contradicting the fact that U = on dB and U(x) —> as 
|a;| — > +oo. □ 

Proof of Theorem 1.8. The proof follows as in the case of Theorems 1.6 and 1.7 
once we get, as in the proof of Theorem 1.3, the existence of a direction e G S ,JV_1 
such that U is symmetric with respect to T(e) and that (5.61) holds. 
To get (5.61) we need the following fact: if we have a direction e G S 1 ^" 1 such 
that W e is either strictly positive or strictly negative in f2(e) and the system is of 
gradient type, then 

(5.66) Q e (^,n(e)) > for any ip G C£(fi(e), « ri ). 

This fact is a generalization of Lemma 2.1 in [9] and follows in a similar way. 
Now, starting from the proof of Theorem 1.1 (or of Theorem 1.2) we get a direction 
e G 5' 7V_1 such that Qu(i(>,Q(e)) > (Q e3 (i(j,Cl(e)) > respectively) for any 
V> G C c x (f2(e), R m ). If iy e = in 0(e) then (5.61) is satisfied (Q ea (V>, fi(e)) = 
Qu(ip, 0(e)), by the symmetry) and we are done. If, else, W e ^ we have, as 
in the proof of Theorem 1.1 (Theorem 1.2 respectively) that W e is either strictly 
positive or strictly negative in 0(e). 

Then, applying the rotating plane method, see Proposition 3.9, we get, using the 
same notations, the existence of (3 > such that W@ = in 0(/3) and < in 
0(/3) for any G [0,0). This means, using (5.66), that Q e (ip,Q{0)) > for any 
G [0, 0) and, passing to the limit, Q e (ip,Q(0)) > 0. The symmetry of U with 
respect to T(0) then implies that Q e (ip,D,(0)) = Qu(ip,Q(0)) > and (5.61) is 
satisfied concluding the proof. □ 
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